# Evaluation Tool

SFR-Judge is a series of evaluation models launched by Salesforce AI Research, aimed at accelerating the evaluation and fine-tuning processes of large language models (LLMs) through artificial intelligence technology. These models can perform a variety of evaluation tasks, including pairwise comparisons, single-item scoring, and binary classification, while providing explanations to avoid black-box issues. SFR-Judge has demonstrated exceptional performance in multiple benchmark tests, proving its effectiveness in evaluating model outputs and guiding fine-tuning.

TC-Bench is a specialized tool for assessing the temporal coherence of video generation models. It measures a model's ability to introduce new concepts and perform relationship transformations across different time points by using carefully designed text prompts, corresponding real videos, and powerful evaluation metrics. Applicable to both text-conditional and image-conditional models, TC-Bench facilitates generative frame interpolation. The development of this tool aims to advance video generation technology, leading to higher quality and consistency in generated videos.

This tool aims to assess the generalization ability, trustworthiness, and causal reasoning abilities of the latest proprietary and open-source MLLMs through qualitative research from four modalities: text, code, images, and videos. This is done to increase the transparency of MLLMs. We believe these attributes are representative factors defining the reliability of MLLMs, supporting various downstream applications. Specifically, we evaluated closed-source GPT-4 and Gemini, as well as 6 open-source LLMs and MLLMs. Overall, we evaluated 230 manually designed cases, with qualitative results summarized into 12 scores (i.e., 4 modalities multiplied by 3 attributes). In total, we revealed 14 empirical findings that contribute to understanding the capabilities and limitations of proprietary and open-source MLLMs, enabling more reliable support for multi-modal downstream applications.

Unitxt is an innovative library tailored for generative language models, specializing in customized textual data preparation and evaluation. natively integrated with popular libraries like HuggingFace and LM-eval-harness, Unitxt breaks down the processing workflow into modular components, enabling easy customization and sharing. These components include comprehensive dataset handling definitions for model-specific formats, task prompts, and more. Unitxt-Catalog centralizes these components, promoting collaboration and exploration within modern text data workflows. Beyond a tool, Unitxt is a community-driven platform, empowering users to collaborate on building, sharing, and advancing their data pipelines.

This product is a tool for evaluating the quality of text-to-video generation. It introduces a new evaluation metric called Text-to-Video Score (T2VScore). This score integrates two key criteria: (1) Text-Video Alignment, which examines the faithfulness of the video in presenting a given textual description; (2) Video Quality, which assesses the overall production quality of the video. Furthermore, to evaluate the proposed metric and promote future improvements, the product provides the TVGE dataset, which collects human judgments on 2,543 text-to-video generated videos for these two criteria. Experiments on the TVGE dataset demonstrate that the proposed T2VScore exhibits superiority in providing a better evaluation metric for text-to-video generation.

Openlayer is an evaluation tool designed for your development and production workflows, empowering you to confidently release high-quality models. It provides robust testing, evaluation, and observability, eliminating the guesswork surrounding the adequacy of your prompts. Support includes LLMs, text classification, tabular classification, and tabular regression. Real-time notifications keep you informed of AI model failures, enabling confident deployment.